sqlite: add incremental blob I/O - #65444
Open
arimxyer wants to merge 2 commits into
Open
Conversation
Collaborator
|
Review requested:
|
nodejs-github-bot
added
c++
arimxyer
force-pushed
the
sqlite-incremental-blob-io
branch
from
August 20, 2026 22:00
9feff52 to
e8aa961
Compare
Author
|
The branch has been updated since this was opened. What changed:
Could a collaborator start CI when convenient? This is my first PR here, so it needs the Actions approval in addition to |
node:sqlite has a single BLOB boundary: a whole SQLite value becomes a whole Uint8Array, on read and on write. Reading one byte of a large value therefore allocates all of it, and there is no partial write path at all. Expose the SQLite incremental blob interface as database.openBlob(), returning a BlobHandle with byteLength, read(), write(), reopen(), close(), and Symbol.dispose. Keep the surface to the primitive so that streams, crypto, and compression can compose on top in userland. The public node:sqlite JavaScript API does not expose its sqlite3* handle. A custom loadable SQLite extension can use the existing connection, but absent such native code a userland implementation needs a second connection. Reading an options property can run user code. Each operation reads its options first, rechecks state, and only then captures the buffer size, buffer pointer, and blob size. This prevents accessors from invalidating state or detaching or resizing a buffer after validation. Handle operations are barred from authorizer callbacks. Identifiers containing NUL are rejected, and zero-length transfers still reach SQLite so its range, aborted-handle, and write-permission checks apply. Closing a writable handle can commit and report an error deferred from an earlier write. A close does not repeat an error already delivered, but it propagates a new commit failure after releasing the handle. BlobHandle disposal deliberately propagates close errors. DatabaseSync disposal does too because this change gives database close a new failure path through owned writable handles. Suppressing either failure would report successful disposal after SQLite rolled back the transaction. Session disposal remains unchanged because its separate error behavior is outside the scope of this change. Closed-state checks preserve idempotence. Refs: https://sqlite.org/c3ref/blob_open.html Signed-off-by: Ari Mayer <ari111097@gmail.com>
The int overload of THROW_ERR_SQLITE_ERROR() sets errcode on the error it throws but not errstr, because it passes the string it just looked up as the message and then calls the const char* overload of CreateSQLiteError(), which takes that as a plain message and leaves the errstr slot empty. The int overload of CreateSQLiteError() already sets both, so call that instead. Every other error raised by node:sqlite carries errstr, and the tests for StatementSync already assert it, so the two paths reaching this helper -- applyChangeset() and a close() that reports a blob commit -- were the only ones where it was missing. Signed-off-by: Ari Mayer <ari111097@gmail.com>
arimxyer
force-pushed
the
sqlite-incremental-blob-io
branch
from
August 21, 2026 17:37
e8aa961 to
706eb9d
Compare
arimxyer
marked this pull request as ready for review
August 21, 2026 17:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
database.openBlob(), which returns aBlobHandlefor reading and writing ranges of a single BLOB or TEXT value without materializing it. It wrapssqlite3_blob_open()and the related C API.BlobHandleexposesbyteLength,read(),write(),reopen(),close(), and[Symbol.dispose]. There is no JavaScript implementation layer;lib/sqlite.jsre-exports the binding.The public
node:sqliteJavaScript API currently materializes a whole value and has no partial-write path. A custom loadable SQLite extension can reachsqlite3_blob_open()on the existing connection, so this is possible outside core with native code. The proposed API makes SQLite's primitive available to ordinary JavaScript without an extension-specific bridge or a second connection.Implementation notes
Options are read before state is captured. Reading an options property can run user code. Each operation reads its options, rechecks the handle and database, and only then captures the current buffer length, backing-store pointer, and
sqlite3_blob_bytes(). This covers accessors that detach or resize the buffer or close the handle/database during argument processing.Close reports only a new deferred error.
sqlite3_blob_close()finalizes the internal statement and can repeat its last result code. The handle remembers the last code, so close does not report an error already thrown by a failed operation. A new close-time commit failure is propagated after the handle is released.Disposal propagates close failures deliberately.
BlobHandledisposal andDatabaseSyncdisposal are idempotent when already closed, but otherwise they do not suppress close errors. Either can close the last writable blob handle and therefore be a commit boundary. Suppressing that failure would make ausingscope appear successful after SQLite rolled its writes back. Session disposal is left unchanged because its separate reentrancy-error behavior is outside this pull request.Incremental writes are raw-byte operations. They do not execute SQL
UPDATE, run triggers, or re-check constraints, and arbitrary writes to TEXT can create invalid UTF-8. The documentation calls out those SQLite semantics, the writable-handle transaction lifetime, and the generated-column and virtual-table restrictions.The second commit fixes a pre-existing error-helper bug exposed by the new database-close path:
THROW_ERR_SQLITE_ERROR(isolate, errcode)selected an overload that lefterrstrempty. It remains separate and can be lifted into its own pull request if preferred.Tests
test/parallel/test-sqlite-blob.jscontains 48 tests covering API options, range and integer validation, buffer detachment and resizing, reentrancy, authorizer callbacks, object lifetimes, SQLite restrictions, raw-byte write semantics, large chunked transfers, and explicit and disposal close failures. The blocked-commit cases verify through a fresh connection that the failed commit rolled back while the exception remained observable.On one Linux machine, reading a 512 MiB value from disk as a whole took about 100 ms and about 600 MB RSS; reading it in 64 KiB chunks through a handle took about 40 ms and about 76 MB RSS. This is a single illustrative measurement, not a crossover study. The peak-memory difference is the motivation; the timing is incidental.
Fixes: #65445 Refs: https://sqlite.org/c3ref/blob_open.html
Assisted-by: frontier coding agent (opus-5)